Skip to content

test(continuity): add suites for gateway restart and package upgrade - #3521

Merged
elezar merged 3 commits into
mainfrom
codex/ubuntu-deb-upgrade-test
Nov 20, 2026
Merged

elezar merged 3 commits into
mainfrom
codex/ubuntu-deb-upgrade-test

Conversation

@elezar

@elezar elezar commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add a continuity e2e family that checks state created before a disruption survives it, with two disruptions: a gateway restart and a package upgrade from the latest published release. Scenarios are seed/verify test pairs that run unchanged under every disruption, so a new disruption such as a rollback or a reboot needs no new test design.

RPM upgrade coverage stays in the stacked follow-up #4202.

Related Issue

Refs #2975 — reusable N-to-N+1 upgrade qualification.

Changes

  • Add the continuity testkit family (tests/suites/continuity). A scenario's seed creates state through the CLI, so testkit cleanup leaves it in place, and its verify adopts that state, checks it, and creates something new. The first scenario writes a marker into a sandbox's workspace. Both tests derive names from a run id (CONTINUITY_RUN_ID).
  • Add openshell-test-runner run --disruption <name>, which runs the seed tests, applies the disruption in the guest, waits for the gateway, and runs the verify tests.
  • Let the installer provide each disruption as an executable under /var/lib/openshell-qualification/disruptions/, because how to restart or upgrade OpenShell depends on how it is installed. The runner keeps only the registry of names and checks that the installer provides the one requested before it seeds anything.
  • Add gateway-restart, rendered by the packaged-gateway role for the user that runs the gateway.
  • Add upgrade and the deb-upgrade-source installer. The installer provisions the guest from the latest release's package and its release-tagged images, and stages the candidate. The upgrade step installs the candidate, requires the installed version to equal the candidate's, switches the gateway to the candidate's runtime images, and restarts it.
  • Prepare the baseline in the upgrade row itself, with the prepare-upgrade-source action and a release tag that select resolves once, so a failed download affects only that row and not the shared prepare-integration job. Select the row with upgrade-source, as external drivers are selected.
  • Add an optional test:e2e:continuity label that runs the family on top of test:e2e, and run it in Release Dev and Pre-release. Release Dev and the qualification result need the continuity jobs, like conformance and feature integration, and the qualification summary records them.
  • Document the family, the installer contract, and the label in TESTING.md and CI.md.

Release Notes

Testing

Ran on macOS with HVF on ubuntu-docker-rootful, with the baseline v0.1.9 and the candidate built from ed8a0d98a:

  • continuity with gateway-restart on the deb installer: passes.
  • continuity with upgrade on deb-upgrade-source: passes. The same run failed twice before for the right reasons: a stale candidate whose database migrations were older than the baseline's, and a wrongly tagged candidate image.
  • upgrade against the deb installer, which doesn't provide it: stops before seeding with an error naming the missing executable.
  • openshell-test-runner: clippy -D warnings, rustfmt --check and unit tests pass.
  • The changed workflows parse, test-qualification-summary.sh passes, and an earlier head passed actionlint, with the row selection run with upgrade-source on and off. Existing row and artifact names are unchanged.
  • markdownlint and nix build .#tmachine-config pass.

Not verified:

  • The new CI jobs have not completed yet, including the baseline download and image export, and actionlint was not rerun on this head.

  • The pre-commit hooks were skipped: rust:lint crashes with a clippy internal compiler error in openshell-supervisor-network, which this PR doesn't touch.

  • A gateway that fails to come back after a disruption is reported as a runner error, not a test failure.

  • Checks appropriate to the affected code and behavior pass

  • Unit tests added/updated (if applicable)

  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@copy-pr-bot

copy-pr-bot Bot commented Sep 21, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

@elezar

elezar commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

/ok-to-test 878d186

@elezar elezar changed the title test(tmachine): add Debian upgrade scenario test(tmachine): qualify DEB and RPM package upgrades Sep 21, 2026
@elezar

elezar commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

/ok-to-test c8ab35d

@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch from 3ca9110 to c9f262b Compare September 22, 2026 08:58
@elezar

elezar commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

/ok-to-test c9f262b

@elezar

elezar commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

The upgrade qualification is currently blocked by a product-level persisted-storage compatibility issue between v0.1.0-pre.4 and HEAD.

Both the DEB and RPM scenarios now pass guest readiness, install the retained prerelease, create the source sandbox, and install the candidate packages. After the candidate gateway restarts, it cannot decode the persisted sandbox protobuf. The observed failures report wire-type mismatches for NetworkEndpoint.enforcement and NetworkEndpoint.tls (LengthDelimited versus Varint).

Initial tracing points to the NetworkEndpoint field-type changes introduced in e38d7254e6099a42d6b80329b8e4ef6f0817931f / #3187, between pre.4 and pre.5. A separate investigation is determining whether this was an intentional compatibility break or requires a storage migration or legacy decoder. Until that is resolved, the package-upgrade tests cannot complete their post-upgrade sandbox continuity assertions.

@elezar

elezar commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

The pre.4-to-HEAD storage blocker described above is now addressed by commit 262732e80.

The existing raw-wire compatibility layer now traverses message-valued protobuf maps and converts legacy NetworkEndpoint fields 4–6 from strings to their current enum varints before Prost decoding. The migration runs for both SQLite and PostgreSQL startup and remains active as a read-time fallback. Focused legacy fixtures, unknown-value rejection, idempotency coverage, and the complete openshell-server test suite pass.

The remaining confirmation is the label-gated DEB/RPM qualification run using packages built from this commit.

@elezar

elezar commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

/ok-to-test 262732e

@drew

drew commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

gator-agent

Blocked

Gator is blocked because this pull request is still a draft.

Next action: @elezar, mark the pull request ready for review when the upgrade qualification work is ready for code review. Gator will review the current head in the next watch cycle after that transition.

Gator metadata
  • Head SHA: 262732e80657310a747e82f2d152dc4910c8a779
  • Gator payload: 10
  • Next state: gator:blocked
  • Blocked reason: draft

@drew drew added the gator:blocked Gator is blocked by process or repository gates label Sep 22, 2026
@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch from cbf12c6 to ee1528d Compare September 23, 2026 13:58
@elezar

elezar commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

/ok-to-test ee1528d

@drew

drew commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

gator-agent

Blocked

Gator is blocked because this pull request is still a draft and the current head has merge conflicts with main.

Next action: @elezar, resolve the conflicts with main and mark the pull request ready for review when the upgrade qualification work is ready. Gator will review the resulting head in the next watch cycle.

Gator metadata
  • Head SHA: ee1528d4f0b1355c5176c3018ee572d8a0253657
  • Gator payload: 10
  • Next state: gator:blocked
  • Blocked reason: draft_and_merge_conflict

@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch from 348a5c3 to 5601c2c Compare October 1, 2026 12:07
@elezar
elezar changed the base branch from main to codex/rpm-tmachine-installer October 1, 2026 12:08
@elezar
elezar force-pushed the codex/rpm-tmachine-installer branch from 86a5838 to 236eab4 Compare October 1, 2026 13:45
Base automatically changed from codex/rpm-tmachine-installer to main October 1, 2026 14:40
@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch from fbdb8c1 to febf3be Compare October 5, 2026 14:16
@elezar elezar changed the title test(tmachine): qualify DEB and RPM package upgrades test(tmachine): qualify Debian package upgrades Oct 5, 2026
@elezar
elezar marked this pull request as ready for review October 5, 2026 15:02
@elezar
elezar requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners October 5, 2026 15:02
@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch from e51715c to fe82250 Compare October 8, 2026 11:03
@purp purp added area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. topic:operability Installing/configuring deployments, upgrades, backup/restore, diagnostics, and maintenance. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure. labels Nov 15, 2026
@github-actions github-actions Bot removed the state:stale Inactive item at risk of automatic closure. label Nov 15, 2026
SDAChess
SDAChess previously approved these changes Nov 19, 2026

@SDAChess SDAChess left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch from 295e946 to f2f756f Compare November 19, 2026 14:38
@elezar elezar added test:e2e:continuity Run the continuity suites (gateway restart and upgrade) on PR mirrors test:e2e Requires end-to-end coverage labels Nov 19, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for f2f756f. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@SDAChess SDAChess left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My agent found two issues with keeping the continuity lane advisory. Both can affect the release flow even though continuity is excluded from the qualification result.

@elezar elezar changed the title test(tmachine): qualify Debian package upgrades test(continuity): add suites for gateway restart and package upgrade Nov 19, 2026
@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch 2 times, most recently from f580d45 to d3f14ae Compare November 19, 2026 19:02
…ption

Add a `continuity` e2e family whose scenarios are `seed` and `verify` test
pairs sharing state through a run-scoped id. With `--disruption <name>` the
test runner seeds, applies the named disruption in the guest, waits for the
gateway, and then verifies, so scenarios stay independent of the disruption.

The runner keeps a registry of disruption names. How a disruption is carried
out depends on how OpenShell is installed, so the installer provides it as an
executable under /var/lib/openshell-qualification/disruptions, and the runner
checks that it exists before it seeds anything. The packaged-gateway role
renders gateway-restart for the user that runs the gateway.

The e2e matrix gains a `disruption` column and a `continuity/gateway-restart`
row on ubuntu-docker-rootful with the deb installer.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Add an `upgrade` disruption that installs a candidate Debian package over a
released baseline, switches the gateway to the candidate's runtime images,
restarts it, and fails if the installed package is not the candidate. The
scenarios stay unchanged and run under it like any other disruption.

The `deb-upgrade-source` installer provisions the guest from a released
package and its release-tagged images, and stages the candidate package and
images under /var/lib/openshell-qualification/candidate together with the
`upgrade` disruption that installs them. The packaged-gateway role keeps the
release's own image configuration for a released source.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Add an `upgrade` row to the continuity matrix that installs the candidate over
the latest published release. The baseline is the release's Debian package and
runtime images, staged under artifacts/upgrade by
prepare-integration-inputs when a caller passes `include-deb-upgrade-source`;
callers select the row with `upgrade-source`, as external drivers are.

Add an optional `test:e2e:continuity` label to Branch E2E Checks that runs the
continuity family on top of `test:e2e`, and run it in Release Dev and
Pre-release qualification. It is not part of either release result yet, so a
failure shows in the run without blocking the release.

Document the label, the family and how to run it locally.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar
elezar force-pushed the codex/ubuntu-deb-upgrade-test branch from d3f14ae to 9a30af5 Compare November 19, 2026 20:28
@matthewgrossman

Copy link
Copy Markdown
Member

Design direction for continuity, separate from the approval below. I'd like us to consider it before more scenarios are written against the current shape.

seed/verify exists mostly because testkit cleanup is per-test. That forces the CLI-create + adopt workaround and the CONTINUITY_RUN_ID plumbing. It also can't express multiple disruptions (kill, wait for recovery, assert, kill again) or disruption-specific assertions (an upgrade checking a migration ran, a restart not).

Proposal: add a cx.disrupt(name) to testkit for disruptions that leave the test process alive. It checks the staged executable, runs it, and waits with the existing Gateway::wait_ready. A scenario then becomes one test: create, disrupt, assert. Keep the runner's seed/verify mode as opt-in for disruptions that kill the test process (VM reboot, snapshot restore), where a guest-resident test can't survive. Related: make the disruption registry data-driven so a new disruption doesn't need a runner change.

Not asking for this in this PR. The cost grows with each scenario, so I'd like a follow-up issue before we add more. I haven't checked whether the SDK channel recovers across a gateway restart; that's worth confirming in a prototype.

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving as a first cut so the deb and rpm work can land. The structure is good: scenarios stay disruption-agnostic and the installer supplies the how. The continuity rows pass in CI. Design direction is in my separate comment.

Small notes, none blocking:

  1. A gateway that doesn't come back after a disruption is reported as a runner error. For an upgrade test that is the primary failure, so I'd report it as a test failure.
  2. upgrade_source_tag is the latest release at run time, so the same candidate can pass one day and fail the next. Consider pinning it, or recording the tag in the summary.
  3. Neither this nor #4202 exercises install.sh, which is the path that broke. I'll add install.sh installers on this disruption contract under #5740, so no change needed here.

@elezar

elezar commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

A gateway that doesn't come back after a disruption is reported as a runner error. For an upgrade test that is the primary failure, so I'd report it as a test failure.

Yes, that should probably change. It may be easier to manage / detect with the testkit-based disruption that we'll look into as a follow-up.

upgrade_source_tag is the latest release at run time, so the same candidate can pass one day and fail the next. Consider pinning it, or recording the tag in the summary.

I don't think we should pin it. That means that we'd have to keep it up to date as part of the release process to ensure that we're testing N-1 -> N. I agree that we could record the tag in the summary.

install.sh

Yes, I am aware of this, but we need install.sh installers before we can add this functionality.

Merged via the queue into main with commit 2a565b2 Nov 20, 2026
164 checks passed
@elezar
elezar deleted the codex/ubuntu-deb-upgrade-test branch November 20, 2026 09:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build CI/CD workflows, releases, build targets, packaging, and artifact publication. gator:blocked Gator is blocked by process or repository gates test:e2e:continuity Run the continuity suites (gateway restart and upgrade) on PR mirrors test:e2e Requires end-to-end coverage topic:operability Installing/configuring deployments, upgrades, backup/restore, diagnostics, and maintenance. topic:testing Unit, integration, regression, conformance, E2E tests, coverage, flakes, and test infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants